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Abstract 

Numerical accuracy of floating point computation 
is a well studied topic which has not made its way to 
the end-user in scientific computing. Yet, it has become 
a critical issue with the recent requirements for code 
modernization to harness new highly parallel hardware 
and perform higher resolution computation. To democ¬ 
ratize numerical accuracy analysis, it is important to 
propose tools and methodologies to study large use 
cases in a reliable and automatic way. In this paper, we 
propose verificarlo, an extension to the LLVM compiler 
to automatically use Monte Carlo Arithmetic in a 
transparent way for the end-user. It supports all the 
major languages including C, C++, and Fortran. Un¬ 
like source-to-source approaches, our implementation 
captures the influence of compiler optimizations on the 
numerical accuracy. We illustrate how Monte Carlo 
Arithmetic using the verificarlo tool outperforms the 
existing approaches on various use cases and is a step 
toward automatic numerical analysis. 

1. Introduction 

This paper presents a new compiler tool to assess 
the uncertainties on a scientific code due to the floating 
point (FP) arithmetic. It builds upon the extensive work 
of Parker IIPar97ll and Frechtling IIFL15I on Monte 
Carlo Arithmetic (MCA) for floating point accuracy 
verification. Floating point computations are a model 
of real number computation where a real number is 
rounded towards a floating point number, and some 
arithmetical properties, such as the associativity of the 
sum, are lost. Consequently, the computer numerical 
results are sensitive to the evaluation order of the 
floating point arithmetical operations, the floating point 
precision, and the rounding mode. 


The quantification of the floating point uncertainties 
is important. In the next decade, exascale supercom¬ 
puters will provide the computational power required 
to perform very large scale simulations. For certain 
applications the results of exascale simulations will 
be of such high resolution that experimental measure¬ 
ments will be insufficient for validation purposes. As 
floating point approximations of numeric expressions 
are neither associative nor distributive, the results of 
a numerical simulation can differ between executions. 
As reported by Duff ODuflll . “Getting different results 
for different runs of the same computation can be 
disconcerting for users even if, in a sense, both results 
are correct”. There is a need to have an automatic and 
global approach giving a confidence interval on the 
results taking into account the floating point arithmetic 
effect. 

Numerical verification is a procedure to estimate 
the effect of the floating point model on the accu¬ 
racy of the computed results. It is the first step of 
a rigorous Verification and Validation (V&V) proce¬ 
dure. Several methods exist to perform a numerical 
verification on a numerical code. Kahan, the primary 
architect of the IEEE-754 standard for floating point 
computation, argues in BKahOhll that using extendable 
precision interval arithmetic is almost foolproof. The 
interval arithmetic is an arithmetic defined on sets of 
guaranteed intervals rather than on sets of IEEE-754 
numbers. The numerical verification on a scientific 
code using IEEE-754 floating point numbers consists 
of comparing the results with those obtained on a 
shadow code using interval arithmetic. It requires that 
the results intervals are sufficiently small. If not, the 
computation needs to be performed again by extending 
the precision of the interval arithmetic. Unfortunately, 
even if it guarantees the result, interval arithmetic 
typically produces overly pessimistic bounds as it does 










not take into account the round-off error compensation 
when using the rounding mode to the nearest. Some 
numerical algorithms need to be adapted when using 
interval arithmetic. For example, the Newton-Raphson 
method needs to be modified in order to obtain conver¬ 
gence under interval arithmetic IIRev03ll . Consequently, 
from an industrial point of view, it is only possible to 
use extendable precision interval arithmetic on specific 
numerical algorithms and not on a whole scientific 
code. 

An alternative is to compute stochastic confidence 
intervals on the results by applying random pertur¬ 
bations on the numerical operations. For example, 
the Discrete Stochastic Arithmetic implemented in 
the CADNA library peiturbates computations by ran¬ 
domly changing the rounding mode. Discrete Stochas¬ 
tic Arithmetic is based on CESTAC developed by 
Vignes in 1974. CADNA is a powerful numerical 
debugger tool which has been used to solve real 
problems. Nevertheless, CADNA has some limitations. 
First, Chatelin and Parker BChaSSI . 0Par97ll show that 
the CESTAC assumptions are not always verified when 
performing numerical analysis, which can introduce 
errors in CADNA estimations. Second, using CADNA 
requires manually modifying the original program 
sources to use special CADNA types. Eor large code 
bases, this process can be costly. 

In this paper, we propose the following contribu¬ 
tions: 


• Verificarlo, a new LLVM compiler tool to au¬ 
tomatically use the Monte Carlo arithmetic in 
place of the IEEE-754 EP. Verificarlo operation 
is transparent for the user and does not require 
manually modifying the source code. 

• A set of experiments to validate the automatic 
MCA approach using verificarlo and compare 
it to the state-of-the-art MCA approach us¬ 
ing CIL IIPL15I and CESTAC approach using 


CADNA |Vig04|. 


This paper is organized as follows. Section 
presents stochastic arithmetic for numerical verifica¬ 
tion. Section introduces the verificarlo tool, its ad¬ 
vantages and limitations, and compares it to other 
approaches. Finally, section proposes a set of ex¬ 
periments to validate verificarlo and demonstrate its 
capabilities. 


2. Probabilistic methods to check the float¬ 
ing point accuracy 


accuracy: the Monte Carlo Arithmetic (MCA) and the 
Discrete Stochastic Arithmetic (DSA). 

2.1. Monte Carlo Arithmetic (MCA) 

Monte Carlo Arithmetic (MCA) tracks rounding 
and catastrophic cancellation errors at a given virtual 
precision t by applying randomization to input and 
output operands. MCA makes no assumption about 
the round-off error distribution and produces unbiased 
random round-off errors. 

It forces the results of floating point operations to 
behave like random variables. This turns executions 
into trials of a Monte Carlo simulation allowing statis¬ 
tics on the effects of rounding error to be obtained 
over a number of executions. This section succintly 
summarizes MCA, for a full presentation see IIPar97l . 

To model errors on a FP value x at virtual precision 
t, Parker proposes the following function: 

inexact{x) = x + (1) 

where is the exponent of the FP value x and ^ is 
a uniformly distributed random variable in the range 

[-1 11 
L 2’ 2 \- 

Each floating point operation a; o y is transformed 
into a MCA EP operation using one of the following 
modes: 

• RR: Random Rounding, which tracks rounding 
errors by introducing an error in the outbound 
value. 


X oy ^ round{inexact{x o y)) 

• PB: Precision Bounding, which tracks catas¬ 
trophic cancellations by introducing errors in the 
inbound values, 

a; o y —)• round{inexact(x) o inexact{y)) 

* Pull MCA: Monte Carlo Arithmetic with inbound 
and outbound errors. 


xoy —)• round{inexact{inexact{x)oinexact{y))) 


When the exact solution x of a problem is known, 
we can measure the number of significant digits s in 
base (3 by computing the magnitude of the relative 
error between the approximated value x and the exact 
value X using the following formula 


s = 



( 2 ) 


The aim of this section is to briefly present two 
probabilistic methods used to check the floating point 


Parker extends this definition to MCA and 
shows 0Par971 p. 23] that the total significant digits 


















for a set of MCA results at virtual precision t is given 
by the magnitude of the relative standard deviation 

s' = - log^ - (3) 

In this formula, fi and cr are the mean and the standard 
deviation of the result distribution. Unfortunately, the 
exact distribution of results is unknown, but it can 
be empirically estimated by using a large number of 
Monte Carlo trials. Indeed for a large number of trials, 
s' « — log^ ?, where fi and a are the sample mean 
and sample standard deviation. 

The metrics given by equations and will be 
used in section to evaluate our outputs and compare 
to other approaches. 

2.2. Discrete Stochastic Arithmetic (DSA) 


Discrete Stochastic Arithmetic (DSA) is based on 
the CESTAC method. The CESTAC method is a pi¬ 
oneer work in the domain of the random computer 
arithmetic 0VLP74II . The ingenious idea is to randomly 
change the rounding mode of a floating point (EP) 
computation to estimate its accuracy. Eor debugging 
purposes, DSA has made the choice to carry out a sin¬ 
gle program in which each EP operation is performed 
N times with a rounding mode towards plus or minus 
infinity. Eor each sample, the rounding mode is ran¬ 
domly chosen. There is thus a probability Pn = 
that all the N samples compute a EP operation with 
the same rounding mode. The number of significant 
digits is computed by using the Student distribution. 
DSA also redefines relational operators. A full review 
of DSA is provided in | Vig04| . 

The CADNA library is an implementation of Dis¬ 
crete Stochastic Arithmetic (DSA) with A = 3 sam¬ 
ples. The first two samples compute each EP operation 
with a random rounding mode whereas the last one 
uses the rounding mode not used by the second sample. 

The validation of CESTAC method and DSA is 
based on a probabilistic first order model. It has been 
established by considering that elementary round-off 
errors of the EP arithmetic operations are random 
independent, centered and uniformly distributed vari¬ 
ables. Kahan has formulated strong objections to this 
assumption by proposing in IIKah96l the following case 
study: 


Khx{dx) = cf{x + dx) — rp{x) 

^ 3(a:-2.0) * ((x-5.0)2-h4) 

~ x+ix- 2.0)2((x -16)2 0) 


( 4 ) 

( 5 ) 


622.0 - a;(751.0 - a:(324 - a;(59.0 - 4.0x))) 
“ 112 - x(151.0 - a;(72.0 - a:(14.0 - a;))) 

(6) 

with : 

. X = 1.60631924 

• dx = ie,i varying from 1 to 300 with step 1, e = 

2-53 

Figure presents the evaluation of equation a by 
using IEEE-754 double precision floating point num¬ 
bers. On one hand, the stripped patterns in figures [T] 
and 12] show that IEEE-754 and CADNA results are 
not uniformly distributed random variables. On the 
other hand, the Monte Carlo Arithmetic introduced 
in section [2.1 1 permits to obtain an independent and 
identically distributed (iid) centered uniform sample 
as shown in figure 

Chesneaux and Vignes argue in IICV88I that even if 
the independent, centered, and uniform assumption is 
not satisfied, the CESTAC method is able to correctly 
estimate the number of significant digits with a proba¬ 
bility of 95%. Nevertheless, Chatelin 0Cha88ll indicates 
that CESTAC’s confidence cannot be greater than 5% 
under various conditions which are shown to be often 
met in practice and Parker IIPar97l explains how MCA 
can overcome these limitations. 

It is important to notice that DSA and MCA also 
differ from a methodological point of view. DSA uses 
a synchronous approach where the user incrementally 
fixes the numerical instabilities reported by CADNA. 
DSA is based on a first order model: some operations 
such as unstable division or unstable multiplication 
may invalidate the model. The user must correct these 
unstable EP operations before CADNA can estimate 
accurately the number of significant digits. Therefore 
DSA is well suited to perform numerical debugging to 
correct numerical instabilities as done in IISJDC07I . 


3. Verificarlo: A software for automatic 
Monte Carlo Arithmetic analysis 


As previously discussed in section 2.1 Monte Carlo 
Arithmetic is a powerful framework to understand 
the numerical stability of a function or program. To 
encourage its wide adoption by the community we 
have developed verificarlo, a tool for automatic MCA 
analysis of C, C-H- and Fortran programs. Verificarlo 
builds upon the LLVM Compiler IILA04I project and 
the MCALIB IIFL15L It takes as input a source code 
project and compiles it with a special instrumentation 
pass that replaces all floating point operations by their 
MCA counterpart in MCALIB. The instrumentation 
can be applied to the whole program or only to a 
function of interest. 
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• DSA run 1 * DSA run 2 DSA run 3 


• MCA run 1 * MCA run 2 MCA run 3 



Figure 1; Eq. [fusing IEEE-754 DP 


Eigure 2: Eq. [fusing CADNA 


Eigure 3; Eq. using verificarlo 


Two previous approaches for automatic MCA sim¬ 
ulation have been proposed. Yeung et al. OYYLllI 
implement MCA at the hardware level through special¬ 
ized EPGA co-processors. While providing low over¬ 
head, this approach is impractical because it requires 
specialised hardware not available to the practitioner. 

Erechtling et al. flFL15ll leverage source-to-source 
rewriting of floating point operations through the CIL 
tool for program transformation IINMRW021 . The first 
drawback of using CIL is that analysis is limited to 
C programs. The second and main drawback with 
source-to-source rewriting is that the instrumentation 
happens before and may hinder the compiler optimiza¬ 
tions. That means that the floating point operations 
in the MCA binary and in the original binary may 
be different. What is tested is not always what will 
be executed because CIL cannot capture the effect of 
compiler optimizations on numerical errors. 

To tackle these issues, verificarlo instruments the 
floating point at the optimized Intermediate Represen¬ 
tation level (IR). First, because the IR representation 
is independent of the source language used, verificarlo 
can operate on any source language supported by 
LLVM that includes C and C-H- through clang and 
Fortran through dragonegg. Second, the instrumenta¬ 
tion pass is done after all the other front-end and 
middle-end optimization passes (which include all the 
floating point optimizations such as -ffast-math 
or -freciprocal-math). 

Verificarlo is available at http://www.github.com/ 
veriflcarlo/veriflcarlo under an open source licence. 
Verificarlo computes Monte Carlo arithmetic using a 
modified version of MCALIB. One notable differ¬ 
ence is that our version of MCALIB replaces the 
standard libc pseudo-random generator with Mersenne 
Twister IIMN98bll . This provides two benefits: first for 
user programs using the libc rand function, having 
a separate generator avoids seeding collisions. Sec- 


version 

samples 

total time (s) 

time/sample (s) 

original program 

1 

.056 

.056 

CADNA 

3 

2.93 

.097 

MCALIB 

128 

1184.02 

9.25 

verificarlo 

128 

834.57 

6.52 

verificarlo 16 threads 

128 

54.39 

.42 


Table 1: Performance in seconds for the numerical 
analysis of the compensated sum algorithm detailed 
in section 4.1 on an array of size 1000000. All the bi¬ 


naries were compiled using -OO. The experiment was 
performed on a 16-core 2-socket Xeon E5@2.70GHz 
with 20Mb L3 cache per socket and 64Gb of RAM. 


ond, Mersenne Twister is a robust random number 
generation in the context of Monte Carlo simula¬ 
tions IIMN98all . 

One disadvantage of MCA is that it requires a large 
number of samples compared to DSA and is therefore 
more costly. Table compares the cost of running 
a numerical analysis with CADNA, MCALIB and 
verificarlo. Verificarlo and MCALIB are significantly 
slower than CADNA. The first reason is that to be 
accurate they require a higher number of samples. The 
second reason is that both MCALIB and verificarlo 
use the MPFR llFHL~*~07ll library to compute MCA 
samples. Performing high precision computations with 
MPFR is more costly than changing the rounding 
mode. 

Fortunately, verificarlo supports massively parallel 
execution out of the box. The high overhead can be 
mitigated by concurrently measuring the MCA sam¬ 
ples. Our tests show an ideal scalability thanks to the 
embarrassingly parallel nature of Monte Carlo. In con¬ 
trast, CADNA parallelization does not scale 0JLC13II 
because it requires explicit synchronization between 
processes. 























4. Experimental results 

This section presents four case studies to illus¬ 
trate floating point accuracy verification using veri- 
ficarlo and its benefits compared to other state-of- 
the-art approaches: CADNA C 1.1.9 OLCJIOI and 
CILh-MCALIB. 

The first case study evaluates the numerical er¬ 
ror in a compensated sum algorithm using CADNA, 
CILh-MCALIB, and verificarlo. Among the three tools, 
only verificarlo is able to capture the effect of compiler 
flags on numerical errors. 

The second case study deals with the solving of a 
linear system Ax = b proposed by Kahan in IIKah66l . 
The matrix A is ill-conditioned which can reduce the 
number of significant digits. This case study demon¬ 
strates how verificarlo using MCA can estimate the 
number of significant digits. The resolution is done 
by using the sophisticated LAPACK routines. It has 
not been possible to use CADNA as it requires to 
manually change the source code of the LAPACK 
library. Verificarlo gives an estimation of the number 
of significant digits close to the number of significant 
digits between the IEEE-754 computing and the exact 
solution. 

The third case study deals with unstable branching. 
In this case, CADNA is too pessimistic as it estimates 
that the numerical result has no accurate digits whereas 
verificarlo finds a number of significant digits close to 
the number of accurate significant digits between the 
IEEE-754 computing and the exact value. 

The fourth case study deals with the management 
of a counter. The comparison between the IEEE-754 
double precision computing and the exact solution 
shows that the numerical result is a numerical noise 
having no significant digits. Unfortunately for this 
case, CADNA is too optimistic as it over-estimates the 
result significant digits whereas verificarlo succeeds to 
estimate that the IEEE-754 DP result is a numerical 
noise. 

4.1. Case study 1: Compensated Summation 

In the following, we demonstrate the importance 
of capturing compiler effects on a standard use case: 
Kahan’s compensated summation algorithm iHi^ 
p. 83] shown on figure The C implementation is 
particularly sensible to compiler optimizations when 
floating point associativity rules are relaxed with 
-f fast-math -03. The compiler uses simple com¬ 
mon subexpression elimination and rewrites line 9 
as sum = sum + f [i] which is the naive non- 
compensated summation. 


1 float sum = f[0]; 

2 float c = 0.0, y, t; 

3 

4 for (int i=l;i<n;i++) { 

5 y = f[i] - c; 

6 t = sum + y; 

7 c = (t - sum) - y; 

8 sum = t; 

9 } 

10 

11 return sum; 


Eigure 4: Kahan compensated summation: with -03 
-ffast-math flags the compiler simplifies and re¬ 
moves the computation of the compensation term c. 



Eigure 5: One thousand MCA RR samples of Ka¬ 
han summation: CILh-MCALIB is unable to capture 
the compiler effect on Kahan’s summation because 
it operates at the source level. On the other hand 
verificarlo operates after compiler optimizations and 
correctly shows that the -OO version is more precise 
than the -03 -ffast-math version thanks to the 
compensation term c. 


Using verificarlo and CILh-MCALIB 0ELI5I we 
measured 1000 sample executions of the Kahan sum¬ 
mation code compiled with -03 -ffastmath and 
-OO. The input array contains 1000 random single 
precision floats in the interval [0,1] and therefore has a 
condition number of 1. Only Random Rounding MCA 
errors were considered in this study. 

Eigure compares the results between verificarlo 



-OO 

-03 -ffast-math 

CADNA 

7 

7 

CIL+MCALIB 

7.3 

7.3 

verificarlo 

7.3 

5.8 


Table 2: Number of significant digits estimated with 
size n = 100000. Verificarlo is the only tool that 
detects that -03 -ffast-math introduces a loss of 
accuracy. 



















flags -OO • * ■ -03 -ffast-math 



2500 5000 7500 10000 

size (n) 


Figure 6; Relative Standard Deviation (^) of Kahan’s 
compensated sum computed on 1000 verificarlo sam¬ 
ples. The compensated -OO version has a constant 
error while the -03 -fast-math error increases as 
O(ey^). 


and CILh-MCALIB. On one hand, CILh-MCALIB is 
unable to detect any difference between the two ver¬ 
sions. Table shows the number of signihcant digits 
predicted by CADNA, CILh-MCALIB and verificarlo 
for an array of 100000 floats. Again, CADNA and 
CILh-MCALIB are blind to compiler optimizations 
because they operate at source level. On the other hand, 
verihcarlo correctly shows the loss of accuracy in the 
-03 -ffast-math version. 

In figure we plot the relative standard deviation 
of verihcarlo’s samples with different input sizes. 
Theoretical error analysis |Hig02 p. 85] shows that 
Kahan’s compensated sum relative error is bounded 
by 2e -h O(ne^) where n is the input size and e the 
computation’s precision. So Kahan’s sum relative error 
is constant for inputs satisfying ne < 1. However the 
relative error of a naive sum grows as O(eyTi) when 
floating point errors are iid with zero mean. We see that 
verihcarlo stochastic error analysis closely matches the 
theoretical error bounds. 

This experiment demonstrates how the late instru¬ 
mentation in verihcarlo helps evaluating the impact of 
compiler optimizations on numerical stability. 


4.2. Case study 2: Resolution of a linear system 

Kahan llKah66l proposes the following linear system 
with a large condition number: 


f 0.2161 0.1441 \ _ f 0.1440 \ 

1.2969 0.8648 j ^ “ V 0-8642 J 


FP arithmetic 


Result 

S 

IEEE-754 single precision 


i(l) = 1.33317912 

0 

(default rounding) 


x(2) = -1.00000000 

0 

IEEE-754 double precision 

*(1) 

= 2.00000000240030218 

9 

(default rounding) 

x(2) 

= 2.00000000359962060 

9 


Table 3: Resolution of equation]^ by using the IEEE- 
754 single and double precision boating point arith¬ 
metic and comparison to the exact solution 


The exact solution of equation]^ is: 



In the context of this case study, we solve equa¬ 
tion |7] using the LAPACK numerical library. LAPACK 
is written in Eortran 90 and provides sophisticated 
routines for solving systems of simultaneous linear 
equations, least-squares solutions of linear systems 
of equations, eigenvalue problems, and singular value 
problems. Table reports the results of the resolution 
using the IEEE-754 single precision and double preci¬ 
sion arithmetic with a rounding mode to the nearest. 

Unfortunately, it has not been possible to use 
CADNA as it requires to modify the source code which 
is difficult and costly to do in a whole numerical library 
such as LAPACK. Eor example. Montan 0MDCL13II 
has developed a modihed version of the LAPACK 
DGEMM routine (matrix multiplication) to efficiently 
use CADNA. In contrast, the use of verihcarlo has 
permitted to implement automatically MCA on the 
whole LAPACK library. The number of samples used 
by MCA in this experiment is set to 1000. Table 
reports the results of the resolution by using the MCA 
single and double precision boating point arithmetic. 

In this example, the estimator given in section |2.1| 
accurately computes the number of signihcant digits. 
Moreover, this estimation does not require knowing 
beforehand the exact solution of the system. 

4.3. Case study 3: Unstable branching 

This section presents the differences between MCA 
and DSA when dealing with branches testing a EP 
value. The following C program is used in this case 
study: 

1 double a,b,c; 

2 a=2.0*sqrt(3.0)/3.0; 

3 b=a*a-a*a; 

4 if (b>=0) 

5 c=sqrt(b)+10.0; 

6 else 

7 c=sqrt(-b)+10.0; 

8 return c; 















FP arithmetic 

Mean value 

Standard deviation 

s’ 

MCA single precision 

x(i) 

= 1.02463705 

ct{x{1)) 

= 6.46717332 

0.0 


x(2) 

= 6.46717332 

( 7 ( a :( 2 )) 

= 9.69851698 

0.0 

MCA double precision 

x(l) 

= 1.9999999992 

o-(a;(l)) 

= 8.4541287415 x 10“® 

8.3 


x(2) 

= -1.9999999988 

( 7 ( a :( 2 )) 

= 1.26782603316 x 10“® 

8.2 


Table 4; Resolution of equation by using the MCA single and double precision floating point arithmetic 


The test on b at line 4 prevents the square root 
computation of a negative number. In the IEEE-754 
standard, the square root of a negative number returns 
NaN (Not A Number). 

Table [^compares the exact value of d, its numerical 
evaluation by using IEEE-754 double precision with 
the rounding mode to the nearest and the numerical 
verification done both by CADNA and verificarlo. 

The estimation of the number of significant digits 
(9.13 decimal digits) is reasonable given that the 
rounded IEEE-754 computation is exact. On the other 
hand, the numerical verification performed by CADNA 
indicates that the result c is a numerical noise having 
no significant digit. 

In verificarlo, each sample execution can follow 
a different branch in the code; executions are in¬ 
dependent. CADNA, unlike verificarlo, works in a 
synchronous mode. Each floating point operation is 
computed three times with different rounding mode 
towards plus or minus infinity. Then a reconciliation 
process is used to select a single branch outcome for 
the three CESTAC traces. Eor the test at line 4, the 
three samples of 6 are &i = —0, &2 = 2.22045 x 10“^® 
and ha = -2.22044 x lO"!®. In this case, CADNA 
reconciliation uses the mean of the three traces, which 
is positive, and concludes that the test is true. Unfortu¬ 
nately, the square root evaluation on the third negative 
sample produces an invalid NaN value. This case study 
shows that CADNA can produce invalid results on 
branch programs. Using CADNA on large code bases 
requires the help of an expert to detect invalid results 
due to branching. In contrast, MCA uses independent 
computing on these samples so no invalid computation 
is performed during this case study. 

4.4. Case study 4: Alternating counter 

In the C code below, a counter c is initialized to 
5 X 10^®. The counter is updated iteratively N times, 
with N = 10®. Eor each update, c is incremented or 
decremented by 10“® depending on the parity of the 
iteration number. 

1 unsigned int i; 

2 double c=-5el3; 


FP arithmetic Result s 


Exact solution 

c = 

-50.0 

- 

IEEE-754 double precision 
rounded to the nearest 

c = 

-0.02460... 

0 

rounded towards — oo 

c = 

-2073773.08... 

0 

rounded towards +00 

c = 

-0.008202... 

0 

rounded towai'ds 0 

c = 

-0.008202... 

0 


Table 6: Comparison between the exact value of c 
and its numerical evaluation by using IEEE-754 dou¬ 
ble precision with the rounding mode to the nearest, 
towards — oo and towards -|-(X) 


3 for (i=0; KIOOOOOOOO; i++) { 

4 if (i%2==0) 

5 c=c+l.e6; 

6 else 

7 c=c-l.e-6; 

8 } 

9 return c; 

Assuming infinite precision, at the end of this pro¬ 
cess the expected exact value of c should be: 

c = -5 X 10®® -b il0®10® - il0®10-® = -50 (9) 

Table compares the exact value of c to its numer¬ 
ical evaluation when using IEEE-754 double precision 
with the rounding mode to the nearest, towards —oo 
and towards -|-c». 

The IEEE-754 arithmetic provides results having no 
significant digits whatever the rounding mode used: 
there is then a strong numerical problem. Table 
reports the results of the numerical verification per¬ 
formed by CADNA and verificarlo and figure the 
evolution of the number of significant bits in the result 
estimated by verficarlo and CADNA. 

CADNA in this case study overestimates the number 
of significant digits of the result. Indeed, the result is a 
numerical noise with no significant digit. Eurthermore, 
CADNA overestimates the number of significant for 
all the program iterations. The overestimation is not 
due to the small number of samples (N=3) used by 
CADNA. The issue here is that each CADNA sample 
performs the arithmetical operations with a rounding 
mode towards -|-(X) or —oo with a probability equal to 
i. As this case study is a linear problem, the expected 











FP arithmetic 

Result 

Significant digits 

Exact solution 

d= 10. 


IEEE-754 double precision 

d= 10. 

s = -t-oo'*’ 

CADNA double precision 

d = @.0 

0 

verificarlo 

100 samples 

d= 10. 

a{d) = 7.255009 X 10““® 

s' = 9.13 


Table 5: Comparison between the exact value of d, its numerical evaluation by using IEEE-754 double precision 
with the rounding mode to the nearest and the numerical verification done both by CADNA and verificarlo. @.0 
means that CADNA found that the result has no significant digits. 

(t) +00 corresponds to the maximum number of representable significant digits, in the case of a IEEE-754 double 
between 15 and 17. 


FP arithmetic 

Result 

Significant digits 

CADNA DP 

c = -0.103 X 10® 
CADNA detects no 

3 

instabilities, only estimated significant digits are printed 

verificarlo DP 
(1000 samples ) 

c = -45.101042 
o-(c) = 44.33 

s' = 0 


Table 7: Numerical verification of the computation of c performed both with CADNA and MCA 


— CADNA DSA verificarlo MCA 



c/3 20- 


O.Oe+00 2.5e+07 5.0e+07 7.5e+07 

Iteration 


1.0e+08 


Eigure 7: Evolution of the estimated significant number 
of bits with the iteration number 


CADNA result should be the mean value of the c 
values computed with IEEE-754 rounding to -|-oo or 
—oo that is to say: 


^ipcadna) — 


-2073773.08...-0.008202... 

2 

-103686 ~ -0.103 X 10^ 


In contrast, the numerical verification performed 
with verificarlo shows that standard deviation is greater 
than the mean value of the MCA samples. It correctly 
indicates that the result computed using IEEE-754 


floating point numbers is a numerical noise. 

5. Limitations and future work 

Verificarlo is a fully automatic tool to instrument 
an application for numerical precision analysis. The 
current version is stable and has been successfully used 
to analyse small and large code bases, yet it is limited 
in some respects. 

As shown in section the verificarlo runtime over¬ 
head is high. This is due to MCA inexact computations 
being performed with MPER. When the desired virtual 
precision is low and known in advance, the overhead 
can be reduced by performing computations using a 
fixed precision implementation (e.g. double, quads) 
and avoiding the MPER abstraction. This improvement 
is scheduled for the next version of verificarlo. 

Unlike CADNA, verificarlo does not support nu¬ 
merical debugging out of the box. In the future we 
would like to include a mode that allows pinpointing 
the exact operation or routine that is to blame for 
a precision loss. We would also like to include a 
statistical post-treatment toolbox to go beyond the 
standard deviation analysis. This toolbox could help 
non-experts understand and interpret the output of the 
MCA analysis. 

Einally, it is important to test the robustness of 
the MCA approach on different classes of numerical 
algorithms such as linear algebra or compensated al¬ 
gorithms and also full-scale real-life applications. 


















6. Conclusion 

The control of the numerical accuracy of scientific 
codes becomes crucial in particular when using HPC 
ressources. It is also necessary to control the floating 
point computation when porting a scientific code on 
another programing language or on different comput¬ 
ing ressources. These tasks raise the need for a tool that 
automatically estimates, without the assistance of an 
expert, the interval of confidence of computed results. 
Verificarlo is the first step toward a fully automatic 
tool. It is based on the Monte-Carlo Arithmetic and 
uses a compiler approach to easily instrument the code 
that is executed. 

Our case studies illustrate the advantages of using 
verificarlo for numerical analysis on scientific codes. 
They show that verificarlo overcomes some method¬ 
ological and technical limitations of the CADNA li¬ 
brary to estimate the numerical accuracy. Verificarlo 
is the first tool to implement MCA arithmetic at the 
intermediate representation level. Unlike CADNA or 
MCALIBh-CIL, this allows to capture the effect of 
compiler optimizations on numerical accuracy. 

Verificarlo is available at http://www.github.com/ 
verificarlo/verificarlo under an open source licence. 
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